Evaluating Deployed Decision Support Systems for Security: Challenges, Analysis, and Approaches

This chapter discusses the difficult problem of evaluating deployed security-focused decision support systems. In a security setting, one can never expect the adversary to cooperate in evaluation, which disallows many kinds of controlled studies. Furthermore, data is typically sparse — there are thankfully relatively few terrorist attacks on major infrastructure in the U.S. Still, evaluating security measures is critical in ensuring efficient allocation of security resources. We discuss a variety of approaches for evaluating such systems, using the deployed ARMOR and IRIS systems as exemplars. Taken as a whole, the evidence supports the effectiveness of these systems, but there are clearly opportunities to improve our methods for assessing the value proposition of all types of security systems.